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Overview 



Many adult students struggle to finance their educations, often contending with work and child 
care expenses in addition to the extra cost of remedial courses. Moreover, there is little need- 
based grant aid to help. This report presents early findings from an evaluation of a program in 
New York City targeted to low-income adults (ages 22 to 35) who need remedial course work. 
Part of MDRC’s national Performance-Based Scholarship (PBS) Demonstration, the program 
operated at Borough of Manhattan Community College and Hostos Community College, both 
part of the City University of New York, in 2008 and 2009. Participating students were eligible 
for either a $1,300 scholarship for two consecutive semesters (totaling up to $2,600) or for 
$1,300 for each of those semesters and one summer term (totaling up to $3,900), if they 
maintained at least part-time enrollment, met attendance benchmarks, and earned at least a “C” 
average across six credits of courses. Scholarships were paid directly to students in increments 
and did not supplant other aid for which students qualified. 

This innovative program, along with those in five other states in the PBS Demonstration, builds 
on lessons from MDRC’s Opening Doors Demonstration in Louisiana, which led to higher rates 
of persistence and credit accumulation. The Louisiana program offered performance-based 
scholarships to low-income parents for two semesters; counselors met with students periodically 
and disbursed the scholarships. Unlike the Louisiana program, the New York PBS program did 
not include counseling and focused on adult students needing remediation. 

The PBS evaluation randomly assigned approximately 1,500 low-income students to one of two 
program groups eligible to receive up to either $2,600 or $3,900 in scholarships, or to a control 
group eligible only for usual financial aid. Comparing outcomes of the combined program 
groups with control group outcomes measures the impact of the scholarship program. Compar- 
ing outcomes for the two program groups speaks to the relative importance of additional 
funding for summer attendance. Early analyses of student transcripts for about 60 percent of the 
total sample suggest that the program: 

• Encouraged more full-time enrollment. Even though the scholarship required on- 
ly part-time attendance, full-time enrollment increased 5.6 percentage points (7.9 
percent) in the first semester and 7.4 percentage points (14 percent) in the second 
semester. Students may have used the funds to purchase more education. 

• Resulted in more students eligible for summer funding registering for summer 
courses. Summer funding increased summer attendance by 7.8 percentage points 
— about a 40 percent increase over the usual 20 percent of students in such courses. 

These findings suggest that incentive scholarships may increase full-time attendance and 
summer enrollment, which may, in turn, benefit low-income adults needing remediation by 
decreasing the time they need to earn degrees or certificates. Future reports will present impacts 
for the full sample on measures not covered here, including student retention, credit accumula- 
tion, graduation and transfer rates, motivation, and use of scholarship monies. 
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Preface 



Community colleges are relatively affordable institutions. During the 2010-2011 academic 
year, the average cost of tuition and fees at community colleges was $2,713, compared with $7,605 
for in-state tuition and fees at public, four-year - colleges. However, once the additional costs of books, 
supplies, transportation, and other basic living expenses are included, the total yearly cost of attending 
a community college can become insurmountable for low-income students. As a result, these students 
often need supplements such as the federal Pell Grant and state financial aid to help pay for college. 
This may be especially true of nontraditional students, who are older, often work, attend school part 
time, and are financially independent of their - parents (and may be parents themselves). Yet their - part- 
time status and work income can limit their access to financial aid programs, which are often de- 
signed for more traditional students fresh out of high school and without work and family obligations. 

Could a performance-based scholarship help such students accelerate their progress toward a 
degree by providing additional financial support and creating an incentive to complete their courses? 
MDRC’s Performance-Based Scholarship Demonstration seeks to answer this question and others by 
testing variations of performance-based scholarships in Arizona, California, Florida, New Mexico, 
New York, and Ohio. 

This report describes early findings from two City University of New York (CUNY) colleg- 
es, Borough of Manhattan Community College and Hostos Community College. Older (22-35 years), 
low-income students who required remediation were randomly assigned to a control group that 
received the colleges’ standard financial aid package or to one of two program groups that were 
eligible to receive the scholarship in addition to their standard financial aid packages. The program 
offered awards of up to $2,600 over two semesters or $3,900 over two semesters and a summer 
semester for maintaining at least part-time enrollment, meeting attendance benchmarks, and earning 
at least a “C” average across six credits of courses. 

The early findings suggest that the program encouraged students to enroll full time during the 
first and second semesters of operation. The modest effect on full-time enrollment is somewhat 
surprising, given that the program required only part-time enro ll ment. In addition, eligibility for a 
su mm er scholarship encouraged more students to enroll in and pass summer courses. Summer 
attendance may be an important step for nontraditional students, as it may help decrease the time 
needed to earn a degree or certificate. 

While these findings are modest, the full story on the effectiveness of performance-based 
scholarships has yet to be told. A future report on these two colleges and the other demonstration sites 
will continue to build evidence about the value of this innovative way to deliver additional financial 
support. 

Gordon L. Berlin 
President 
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Introduction 

This report presents early results from a rigorous evaluation of a performance-based 
scholarship program for low-income adults in community college in New York City who 
require some remedial course work. 1 The evaluation uses an experimental design, similar to 
those used in medical trials, to measure the effects of the program on academic success and 
persistence. Students who consented to participate in the study were randomly assigned to one 
of three groups: 

1 . A program group (known as the PBS group ) that was eligible for a scholar- 
ship of up to $1,300 each semester for two semesters (or up to $2,600); 

2. A second program group (called the PBS plus summer group ) eligible for the 
above scholarship and a performance-based scholarship of up to $1,300 for 
one consecutive summer term (or up to $3,900 total); or 

3. A control group eligible to receive whatever regular financial aid package for 
which the members qualified. 

The three-group design permits the evaluation to examine whether additional summer support 
enhances the effects of the scholarship. The grants were paid on top of any other financial aid 
awarded so that they represent a net increase in financial aid. Scholarships were paid in incre- 
ments to students who met attendance and grade benchmarks. By being structured around 
benchmarks and incremental payments, the grants acted as an incentive that rewarded behavior 
associated with academic success. 

This program, which started in fall 2008, is part of MDRC’s national Performance- 
Based Scholarship (PBS) Demonstration. The PBS Demonstration was launched in 2008 to 
evaluate whether performance-based scholarships are an effective way to improve persistence 
among low-income college students. Performance-based scholarships are need-based, contin- 
gent grants that are paid in addition to other existing financial aid programs. In other words, the 
intervention generally results in students receiving more money. The scholarships are paid to 
students based on their academic performance in the current term, regardless of what happened 
in previous terms. That is, students receive a payment in each term if they meet certain perfor- 
mance benchmarks in that term — and the award of the payment is independent of whether they 
received any performance-based scholarship payments in prior terms. The scholarships are 
designed to be paid directly to students rather than credited to their accounts with their college. 



'The terms “remedial” education and “developmental education” are used interchangeably throughout this 
report. 
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This permits students to use the funds for their most pressing needs, whether those needs are for 
books, car repairs, child care, or for other expenses that have the potential to disrupt studies. 

Evidence from MDRC’s Study of Performance-Based Scholarships in 
Louisiana 

The PBS Demonstration builds on the promising findings from MDRC’s Opening 
Doors study of performance -based scholarships in Louisiana. In that study, which operated 
from 2004 to 2005, two colleges implemented a performance-based scholarship program for 
low-income parents who were predominantly single mothers. 2 The program provided students 
with scholarships of up to $1,000 for each of two semesters (up to $2,000 per student in total), 
paid in increments based on each student's success in meeting key benchmarks: 3 

• $250 upon enrollment (at least half-time, defined as six or more credit- 
hours); 

• $250 after midterms, contingent upon staying enrolled at least half-time and 
earning a “C” average or better; and 

• $500 upon completion of courses, with a “C” average or better across all 
courses (overall, not per class). 

Performance-based scholarships were distributed in addition to other financial aid for 
which students were eligible. The scholarships were designed to help offset some of the 
financial burdens of attending college, including living expenses, while also rewarding academ- 
ic achievement. The Louisiana program also provided dedicated counselors who monitored 
students' performance, verified that the benchmarks were met, and handed out scholarship 
checks to their students. 

The early findings showed that this approach resulted in substantial improvements in 
scholarship recipients' grades, credit accumulation, and semester-to-semester persistence. 4 For 
example, compared with 50 percent of the control group students, 65 percent of the program 
group students registered for courses in their second semesters (for an increase of 30 percent). 



2 The Opening Doors Demonstration tested several interventions to improve student success at community 
colleges. See Scrivener and Coghlan (201 1) for a synthesis of the findings from the various sUtdies. 

3 The $2,000 scholarship in Louisiana eliminated about 19 percent of unmet need. The scholarship pay- 
ment in New York was calibrated to reflect the assumption that $2,600 represents about 22 percent of the 
estimated unmet need for sUidents living away from their parents. 

4 Brock and Richhurg- Hayes (2006); Richburg-Hayes et al. (2009); Barrow et al. (2009). Unfortunately, 
the devastation inflicted by Hurricane Katrina has made it impossible to determine any longer-term effects of 
the program, such as its impact on transfer and graduation rates. 
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Moreover, these positive trends extended to the third and fourth semesters, when most students 
were no longer eligible to receive the scholarships. These effects are among the largest that have 
been detected in random assignment studies of programs designed to increase persistence 
among community college students . 5 

Background on the Performance-Based Scholarship Demonstration 

While the Louisiana Opening Doors study is an important contribution to the literature 
because it is one of only a handful of studies that is able to measure the effect of additional 
financial aid on academic success, it is just one test . 6 The positive findings could result from the 
location of the study in a state with few need-based resources or from the nature of the target 
group of low-income parents (who may be more mature and capable of utilizing the scholarship 
than other students) or from a combination of these two factors. The goal of the PBS Demon- 
stration is to build more evidence about whether performance-based scholarships help at-risk 
students succeed academically and persist at higher rates than they normally would in the 
absence of such an intervention. 

Currently, eight colleges and one intermediary organization across six states are partici- 
pating in the PBS Demonstration: Two of the institutions are the subjects of this report — 
Borough of Manhattan Community College and Eugenio Maria de Hostos Community College 
(called Hostos College in the report) in New York. The other institutions in the demonstration 
are: Lorain County, Owens, and Sinclair Community Colleges in Ohio; the University of New 
Mexico; Pima Community College in Arizona; Hillsborough Community College in Florida, 
and a state-level intermediary in California. While the amount and duration of the scholarships, 
performance criteria, and target groups for the intervention vary among the programs, all 
programs offer an incentive scholarship designed to address the financial needs of low-income 
students that are not fully covered by existing federal and state financial aid programs . 7 

Each state intervention is being evaluated using random assignment. Random assign- 
ment ensures that the motivation and personal characteristics of students in the program group 
and the control group are the same at the beginning of each study; hence, any subsequent 

5 See Brock (2010) for a review of experimental and quasiexperimental evaluations of program interven- 
tions that target low-income college students. 

6 Factors that are associated with financial need, such as a low income, are also associated with a lack of 
academic success, making it difficult to isolate the effect of additional financial aid on student achievement. 
However, those nonexperimental analyses that have been conducted suggest a positive relationship between 
grant aid and persistence. See Bettinger (2004); Choy (2002); Dynarski (2003); and Leslie and Brinkman 
(1987). Two random assignment studies of financial incentives also suggest a positive relationship for a 
subgroup of students. See Angrist, Lang, and Oreopoulos (2009), and Leuven, Oosterbeek, and van der 
Klaauw (2006). 

7 See Richburg-Hayes et al. (2009) for an overview of the programs in each state. 
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differences in educational or other outcomes can be attributed with a high level of confidence to 
the PBS project. In this way, the demonstration is testing whether performance-based scholar- 
ships are an effective way to improve persistence in postsecondary education and success 
among low-income students in different geographic locations with different amounts of monies 
offered over different durations of time. 

Recruitment for the New York study began in fall 2008 and ended in fall 2009. Slightly 
more than 1,500 students were recruited over the three semesters. Students’ participation in the 
scholarship programs is now finished, but the research for the demonstration is still in progress. 
This report presents findings only for the fall 2008 and spring 2009 cohorts — in other words, 
students who were randomly assigned at the beginning of the fall or spring semesters. These 
cohorts account for about 60 percent (882 students) of the full sample. The remainder of this 
report provides additional background information on why the performance -based scholarship 
program in New York is worth testing, describes the implementation of the program at the New 
York colleges, and reports early impact findings. 



The Policy Context for the New York Program 

Over half of all students attending postsecondary institutions — about 55 percent — are 
adults aged 22 and older. 8 Yet financial aid policy continues to best serve “traditional” students 
— 18 to 21 -year-old high school graduates who are financially dependent on their parents and 
pursue degrees full time. As a result of financial aid regulations, adult students often have 
higher levels of unmet need than traditional students. 9 This is in part because the federal aid 
calculations for independent students assume that even though they are now enrolled in college, 
their previous year’s incomes will remain available throughout their courses of study. Further, 
adult students are expected to contribute a large proportion of their incomes toward their 
education despite other financial responsibilities they may bear — for example, the costs of 
caring for dependents or work expenses. 10 In addition to these possible barriers to success, adult 



8 Snyder, Dillow, and Hoffman (2008), Table 183. 

9 The total cost of attendance includes tuition and fees, books, transportation, and room and board. Unmet 
need is the difference between the total cost of attendance, a family’s estimated contribution toward the 
student's education, and awarded financial aid. Independent students typically have higher living expenses 
(including room and board) than traditional students. As a result, their total cost of attendance is higher. 

"’independence may result in a higher Expected Family Contribution (EFC) — the amount the federal 
government estimates that a family is able to contribute to the expenses of higher education. Since the current 
financial aid system allocates a larger proportion of a student’s income than the income of their parents toward 
college expenses, independent students can have higher EFCs than dependent students. Higher EFCs are 
associated with lower amounts of financial aid. Further, the EFC amount can change dramatically with small 
changes in income. This can be particularly problematic if a student receives a small raise. See Long (2007) 
and Choitz and Widom (2003). 
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students may face other financial hurdles unrelated to financial aid policy. For example, many 
adult students require remediation in math, English, or writing before attempting college -level 
work. Since developmental credits typically do not count toward a degree, the need for these 
courses increases the cost of education. Juggling these financial burdens can be difficult, 
particularly for low-income adult students — the very group most at risk of not persisting to 
complete a certificate or degree. 1 1 

Why should one care about the education-related financial struggles of low-income 
adults? One important reason is related to current public policy goals designed to strengthen the 
U.S. economy and its human capital by greatly increasing the number of the nation’s communi- 
ty college graduates. For example, President Obama has articulated a goal of producing 5 
million more graduates from community colleges by 2020, and major philanthropic foundations 
have endorsed similar increases. Given current demographic projections, these goals cannot be 
reached by targeting high school students alone. Moreover, higher education is associated with 
higher income, as college graduates earn substantially more than high school graduates. The 
New York performance-based scholarship program will provide needed information about 
whether this intervention works (or does not work) to increase graduation rates among low- 
income adults who are not fully prepared for college-level work. 

Overview of Financial Aid for New York State Residents 

The primary need-based financial aid program for college students in the United States 
is the federal Pell Grant program, which makes awards to students based on the cost of atten- 
dance at an institution minus the Expected Family Contribution (which takes into account 
available income and assets). From 2003 to 2007, the average Pell Grant award to students was 
just under $2,500, and the maximum award was capped at $4,3 10. 12 Under the Health Care and 
Education Reconciliation Act of 2010, the maximum award will be tied to increases in the 
Consumer Price Index and will increase to $5,975 by 2017. 13 

In addition to the federal Pell Grant, many states provide need-based grants to state res- 
idents who attend postsecondary institutions within the state. New York State’s Tuition Assis- 
tance Program (TAP) — among the most generous state-funded financial aid programs in the 
country — provides up to $5,000 in financial aid to eligible students, though the actual grant 
award cannot exceed the amount of tuition and fees. 14 The program requires that students 



n Kazis etal. (2007). 

12 Shaw (2007). 

l3 Congressional Research Service (2010). 

14 The income limit is $80,000 for married independent students or independent students with dependents, 
and the maximum TAP award is $5,000 or 100 percent of tuition. For single independent students without 
dependents, the maximum TAP award is $3,025 or 100 percent of tuition, and the income limit is $10,000. 



5 




matriculate full time during the freshman year in order to be eligible for a full-time or part-time 
grant in subsequent terms. 15 

The Pell Grant program and TAP provide the foundation of need-based aid for New 
York residents attending postsecondary institutions in the state. However, for several reasons 
adult students have substantially average lower TAP payments than traditional students. 16 First, 
most adult students work full time to support themselves, and this full-time work places them in 
a bind: Their earnings prevent them from getting the maximum amount of TAP available and 
yet their earnings are insufficient to cover the full cost of their education. 17 Second, many adult 
students attend college part time, which results in significantly lower grant amounts. Finally, 
TAP, which provides six semesters of support for an associate’s degree and eight for a bache- 
lor’ s degree, often runs out for students needing high levels of remediation before they are able 
to complete a two-year or four-year degree. 18 This is particularly problematic for adult students 
requiring remediation who can only attend school part time. (See Box 1 for an illustration.) 



Performance-Based Scholarships in New York 

The Colleges 

This study was implemented at Borough of Manhattan Community College (BMCC) 
and Hostos Community College, which are both part of the City University of New York 
(CUNY) system. BMCC is one of the largest community colleges in the CUNY system. In 
2007, it served about 18,462 students, who hailed from all five boroughs of New York City and 
the adjacent suburbs of New Jersey. BMCC’s student body is diverse, without a racial or ethnic 
majority, though black and Hispanic students each represent about one-third of the student 
population. (See Table 1.) Hostos, located in the Bronx, is a smaller institution; in 2007 it served 
about 4,400 students. As also shown in Table 1, Hispanic students represent a little more than 
half of the college’s student population. 



15 To be eligible for Part-Time TAP, part-time students must have been a first-time freshman in the 2006- 
2007 academic year or later, must have earned 12 credits or more in each of two consecutive semesters, and 
must maintain a “C” average. 

16 For TAP eligibility requirements, see New York State Higher Education Services Corporation (2009). 
See, also, notes 14 and 15 above for information on income limits, maximum TAP awards, and eligibility 
requirements for part-time students. 

17 Kazis et al. (2007). 
ls Hilliard (2007). 
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Box 1 



Average Expenses for BMCC Students and Maximum Available 
Need-Based Financial Aid, Academic Year 2007-2008 

A low amount of financial aid is often a barrier to academic success and persistence for 
low-income adults because it results in high levels of unmet need. Unmet need is calcu- 
lated as follows: (total cost of attendance + family’s estimated contribution) - awarded 
financial aid. 

The table below illustrates how limitations on financial aid affect adult students: As the 
table shows, tuition and fees at Borough of Manhattan Community College (BMCC) to- 
taled $3,068 during the 2007-2008 academic year, while the average cost of books and 
supplies was $938. However, the total cost of attendance for independent students living 
away from their parents was $16,396 because of the students’ relatively high living ex- 
penses (an average of $12,390). 

For single independent students with children, the maximum New York State Tuition 
Assistance Program (TAP) award would have been $3,068 in the 2007-2008 academic 
year — the amount of tuition and fees. However, most older students would have received 
far less than this amount because they often would have started college part time or would 
have interrupted their studies at some point. Even if such students were allotted the maxi- 
mum TAP award and the maximum Pell Grant amount of $4,310 (the maximum amount 
in the 2007-2008 academic year), they would have been left with upwards of $9,018 in 
uncovered expenses. This large gap between need and financial support is fairly common 
among low-income adults attending community college. 



Expenses and Financial Aid at BMCC, 2007-2008 



Category 


Amount 


Total cost ($) 


16,396 


Tuition and fees 


3,068 


Books and supplies 


938 


Living expenses 


12,390 


Maximum need-based financial aid ($) 


7,378 


Maximum TAP 


3,068 


Maximum Pell Grant 


4,310 


Uncovered expenses' 1 (S) 


9,018 



SOURCES: Data are taken from the Integrated Postsecondary Education Data System (IPEDS). Financial aid 
data are the authors’ calculation. The source for the calculation in note a, below, is 
http://www.finaid.org/calculators/faaefc.phtml (accessed January 17, 2011). 

NOTES: “The illustration does not include an estimated amount for Expected Family Contribution (EFC). 
Assuming that the student is independent of his or her parents, has a child, and has income at least 1.5 times the 
poverty threshold for a family of two in 2008 (income of $22,260), the EFC would be about $774. The unmet 
need equals the uncovered expenses ($9,018) minus the EFC, or $8,244. 
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The Performance-Based Scholarship Demonstration 
Table 1 

Selected Characteristics of Colleges Participating in the Performance-Based 
Scholarship Demonstration in New York 



Borough of Manhattan and Hostos Community Colleges 



Characteristic 


BMCC 


Hostos 


Total students 


18,462 


4.416 


Enrollment (%) 


Full time 


63.0 


65.2 


Part time 


37.0 


34.8 


Gender (%) 


Male 


40.3 


28.6 


Female 


59.7 


71.4 


Age a (%) 


Under 25 years 


66.0 


58.4 


25-29 years 


16.1 


16.0 


30 years and over 


17.9 


25.6 


Race/ethnicity (%) 


Hispanic 


31.2 


55.0 


White 


13.8 


2.5 


Black 


33.7 


28.3 


Asian or Pacific Islander 


11.0 


2.7 


Other b 


10.3 


11.5 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Integrated Postsecondary 
Education Data System (IPEDS). 

NOTES: Data on undergraduate degree-seeking students are from fall 2007. Missing values are not included in 
individual variable distributions. 

Distributions may not add to 100 percent because of rounding. 

“Data on age are based on the entire undergraduate student population, 
includes nonresident aliens, American Indians, and Alaskan Natives. 



Target Population 

The New York study was designed to help low-income adults who require some 
developmental course work. Specifically, in order to be included in the demonstration’s target 
population, students who entered the study had to: 

• Be between the ages of 22 and 35 

• Be living away from their parents 

• Be eligible for a federal Pell Grant 






• Have tested into (and not yet passed) at least one developmental/remedial 
course 

• Be enrolled in at least six credit or contact hours at study intake 

These criteria, along with the requirements of TAP, largely ensure that students in the 
demonstration’ s study sample had among the highest levels of unmet need of students in their 
colleges. 

Program Model 

All program group students were eligible for the scholarship of up to $1,300 for two 
consecutive semesters (excluding summer), for a total of up to $2, 600. 19 The scholarship was 
paid directly to students in increments as the student met three key benchmarks: 

• Initial: $200 after registration (the first benchmark) 20 

• Midterm: $450 upon continued midsemester enrollment (the second 
benchmark) 21 

• Final: $650 after receiving a grade of “C” or better (or a Pass in remedial 
courses) in at least six credits or equated credits 22 (the third and final bench- 
mark) 

The first and second payments are designed to offer an incentive to maintain a mini- 
mum level of attendance, while the last payment is designed to offer an incentive for perfor- 
mance. In an effort to reward progress, the program allows students who miss the midterm 
payment to recoup that payment at the end of the semester if they meet the final performance 
benchmark. Also, as noted, eligibility for a scholarship in the subsequent semester is indepen- 
dent of performance during the previous semester. Thus, students have a new opportunity to 
earn the scholarship each semester. 



1 financial aid regulations prohibit certain students from receiving financial aid in excess of their need. In 
these instances, federal work-study aid or loans may be displaced for the performance -based scholarship. 
Given the high levels of unmet need among the target group, less than 1 percent of students were affected by 
these regulations. 

2,) In order to receive the initial payment, each student had to have paid his or her tuition or established 
payment plan. As a result, the demonstration's definition of registration is more stringent than the usual 
definition, since students who have reached the point of paying or establishing a payment plan are more likely 
to attend classes than students who have only registered for classes. 

“'Students received this payment if they attended at least once in the first three weeks and at least once 
during the fourth or fifth weeks of the semester. 

22 Equated credits are awarded in remedial courses. These credits do not count toward a degree or certificate. 
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Box 2 



PBS Group Versus PBS Plus Summer Group 

The Opening Doors Louisiana study shows that the performance-based scholarship program 
increased academic persistence. However, findings from the study also revealed that the 
increase in persistence was not constant across cohorts. (The term “cohort “refers to a group 
of students randomly assigned during a particular semester.) For example, the results suggest 
that the impact was largest when the second semester occurred in the summer — a term con- 
sidered by some to be a nontraditional semester and one that students often bypass. While the 
results suggest that paying for persistence may go further if pay is granted for summer atten- 
dance, they are not definitive. This is because the effect of summer attendance cannot be 
cleanly disentangled from the effect of the payment in the Louisiana study. That is, larger 
impacts in summer are mainly evident when summer is the second (scholarship-eligible) 
semester. In the cases in which summer was not the second semester, results were mixed. 
Thus, the effect of the summer attendance cannot be cleanly disentangled from other possible 
effects associated with whether the payment was the second in a sequence. 

Because the mechanism driving the large impacts during the summer semester in Louisiana 
is unclear, the New York study was designed to analyze directly whether the effects of the 
intervention are enhanced by offering funding for summer attendance. This was done by 
randomly assigning half of the program group to receive up to a $2,600 grant over two seme- 
sters and half to receive up to a $3,900 grant over two semesters and a summer term. By 
including a bonus for summer attendance to all cohorts (independent of whether summer was 
the cohort’s second semester), the three-way design allows researchers to determine whether 
additional funding for summer attendance greatly enhances the effect of the scholarship. 



Encouraged by the compelling findings on summer attendance from the Louisiana pro- 
gram, researchers incorporated a summer component into the model used for the New York 
performance-based scholarship. (See Box 2 for more information.) In order to test the effect of 
the scholarship on summer attendance, half of the program group was randomly assigned to be 
eligible for an additional $1,300 during the summer semester, for a total scholarship amount of 
up to $3,900 across three semesters. Given that summer sessions are shorter than a regular 
semester, the summer disbursement schedule differed slightly from the schedule for a regular 
semester just outlined. 

Another difference between the structure of the scholarships for regular and summer 
semesters is that part-time status during summer sessions was met when students took at least 
three credits, rather than the six credits that are required for part-time status during fall and 
spring terms. In addition, unlike the scholarships for the regular semesters, the summer scholar- 
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ships varied in amount depending on whether they were for part-time or full-time work. During 
the summer, eligible program group students were paid if they met the following benchmarks: 

• Initial: $200 after registration and 

• Final: 



— Part-time benchmark: $450 after receiving a grade of “C” or better 
(or a Pass in remedial courses) in three to five credits or equated 
credits 23 



OR 

— Full-time benchmark: $1,100 after receiving a grade of “C” or better 
(or a Pass in remedial courses) in at least six credits or equated 
credits. 

Implementation of PBS 

The process for implementing the PBS program at both BMCC and Hostos had three 
phases — inviting students to participate, enrolling students into the program, and monitoring 
students’ progress and the disbursement of scholarships. (Figure 1 shows the basic random 
assignment process associated with enrollment.) For all of these activities, BMCC and Hostos 
designated staff to manage and implement the program. At BMCC, the Director of Financial 
Aid was responsible for the oversight of all aspects of program implementation, while at 
Hostos, the Assistant Dean for Student Development and Enrollment Management managed the 
recruitment and enrollment process, and the Director of Financial Aid managed the monitoring 
and payment process. In addition to the senior staff who had overall management responsi- 
bilities, both colleges had coordinators who were responsible for the day-to-day implementation 
of their projects. 

The staff at BMCC and Hostos committed to a systematic recruitment plan, which in- 
cluded mailing invitation letters to all students known to meet the eligibility criteria, creating an 
online reservation system, and conducting follow-up phone calls to students to encourage them 
to attend an orientation session to hear about the study and then to enroll in it. This recruitment 
effort was very successful at both colleges. 



23 The total amount a student could receive for three credits in the summer is half the full amount that could 
be received for six credits. 
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The Performance-Based Scholarship Demonstration 
Basic Random Assignment Process for the Demonstration in New York 

Figure 1 

Borough of Manhattan and Hostos Community Colleges 
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In collaboration with MDRC, each college also developed an effective process for 
enrolling students in the study. The process began with a presentation, which was followed by a 
question-and-answer period, and it ended with students completing an online survey, which 
asked them for basic demographic information and other basic information, for the research. 

MDRC then randomly assigned students to the program group that was eligible to re- 
ceive the performance-based scholarship or to the control group that received whatever financial 
aid package members would have received in the absence of the program. In response to 
concerns of college staff that randomly assigning students at the orientation sessions could 
result in strong emotional responses (since some students would be given an opportunity to earn 
a scholarship, while others would not), random assignment was conducted after the sessions. 
Shortly after random assignment, students were mailed letters informing them of the research 
group to which they had been assigned. 

BMCC and Hostos staff monitored the progress of program group students to determine 
whether the students were eligible to receive payments based on each benchmark. Staff at both 
schools paid close attention to the payment process and were careful to note when students 
missed payments due to ineligibility. Given that the New York program did not have a counsel- 
ing component, the scholarship payment staff restricted their interactions with students to 
answering questions related to financing their educations — for example, questions about when 
payments would be received or about what other financial aid options were available to students 
in the event that they did not meet the scholarship benchmarks. 24 



Data Sources 

Several data sources were used for the analyses presented in this report. First, as shown 
in Figure 1 and mentioned above, students completed a baseline questionnaire that asked them 
for demographic and other background information before they were randomly assigned to the 
program or control groups. Baseline data are used to describe the sample, and in future reports 
these data may be used to make statistical adjustments and define subgroups of sample mem- 
bers. Second, MDRC obtained performance-based scholarship payment records from BMCC 
and Hostos. Findings on the scholarship payments are presented to describe program implemen- 
tation. Finally, BMCC and Hostos provided student-level transcript data for the sample mem- 
bers in the study. These data, which encompass measures like enrollment status, credits at- 



24 Given that the scholarship program at BMCC was completely managed by the financial aid office, stu- 
dents were not likely to receive any additional guidance or counseling related to the program. There is 
suggestive evidence that students at Hostos received additional ad hoc advising, since the coordinator was part 
of the student services division. 
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tempted and earned, and grade point averages (GPAs), were used to provide a detailed look at 
sample members’ performance in college. 



Characteristics of the Performance-Based Scholarship Sample 

This report presents the early findings for the first two cohorts of the research sample — 
the fall 2008 cohort and the spring 2009 cohort, which together comprise about 60 percent of 
the full sample of students in the New York sites (882 students out of the full sample of 
1,502). 25 About half of these students were in the program group and half in the control group. 
As discussed, in addition to being eligible for the two-semester scholarship, about half of the 
program group was randomly selected to be eligible for a summer scholarship. 

Table 2 presents the demographic characteristics of the students who enrolled in the 
PBS study at BMCC and Hostos for the fall 2008 and spring 2009 semesters. The first column 
shows characteristics for the full sample, and the subsequent columns present this information 
separately for each college. 

The sample at both colleges was primarily female. About half of the sample was be- 
tween the ages of 22 and 26 and the majority was unmarried (75.3 percent). Almost half of the 
sample members had one or more children. 

Nearly all of the PBS sample members were financially independent of their parents. 
Slightly more than half of the sample members were working at the time they enrolled in the 
study. Of those who were working, about two-th ir ds worked an average of 21 or more hours per 
week. When asked why they enrolled, about half of the sample members said they wanted to 
earn an associate’s degree, and 45.2 percent wanted to transfer to a four-year college (93.7 
percent gave one or more of these responses; this finding is not shown in the table). Close to a 
third of the sample members said that they were the first person in their family to attend college. 

While the samples at both colleges shared many similarities, there are also some charac- 
teristics that make the samples distinct. At Hostos, nearly 72 percent of sample members were 
Hispanic, and almost 80 percent reported that they regularly spoke a language other than 
English in the home. Compared with the Hostos sample, the sample at BMCC had students who 



25 In this study, students were randomly assigned just prior to the start of a semester and were then consi- 
dered to be part of that semester’s cohort. In other words, a sUident randomly assigned near the start of the fall 
2008 semester would be considered a member of the fall 2008 cohort. 
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The Performance-Based Scholarship Demonstration 
Table 2 

Selected Characteristics of Sample Members at Baseline, by Site, 
Fall 2008 and Spring 2009 Cohorts 



Borough of Manhattan and Hostos Community Colleges 



Characteristic 


Full 

Sample 


BMCC 


Hostos 


Gender (%) 


Male 


30.8 


33.2 


18.6 


Female 


69.2 


66.8 


81.4 


Age (%) 


22-26 years 


54.8 


56.9 


43.6 


27-30 years 


26.1 


25.5 


29.3 


3 1 years and over 


19.2 


17.7 


27.1 


Average age (years) 


26.7 


26.6 


27.5 


Marital status (%) 


Married 


17.0 


15.4 


25.7 


Unmarried 


75.3 


77.2 


65.0 


Missing 


7.7 


7.4 


9.3 


Race/ethnicity a (%) 


Hispanic 


43.9 


38.5 


71.9 


White 


7.1 


8.3 


0.7 


Black 


37.1 


39.4 


25.2 


Asian or Pacific Islander 


9.2 


10.7 


1.4 


Other 


2.7 


3.1 


0.7 


Number of children (%) 


0 


52.1 


54.3 


39.9 


1 


25.7 


25.0 


29.7 


2 


14.3 


13.5 


18.8 


3 or more 


7.9 


7.2 


11.6 


Among sample members with children 


Average age of youngest child (years) 


4.3 


4.2 


4.5 


Household receiving any government benefits (%) 


35.8 


33.8 


46.4 


Missing 


12.1 


12.1 


12.1 


Financially dependent on parents (%) 


1.4 


1.2 


2.1 


Missing 


5.4 


4.4 


10.7 


Currently employed (%) 


56.8 


56.7 


51 A 



(continued) 



15 






Table 2 (continued) 



Characteristic 


Full 

Sample 


BMCC 


Hostos 


Among those currently employed 

Number of hours worked per week in current job (%) 


1-10 hours 


6.7 


6.4 


8.1 


11-20 hours 


24.4 


23.4 


29.7 


21-30 hours 


21.0 


20.0 


27.0 


31-40 hours 


44.8 


47.0 


32.4 


More than 40 hours 


3.1 


3.2 


2.7 


Average hourly wage at current job ($) 


11.2 


11.4 


10.5 


Highest grade completed (%) 


10th grade or lower 


16.9 


17.1 


15.7 


1 1th grade 


12.9 


12.8 


13.6 


12 th grade 


64.9 


65.2 


62.9 


Missing 


5.3 


4.9 


7.9 


Diplomas/degrees eamed b (%) 


High school diploma 


64.1 


64.4 


62.5 


General Educational Development (GED) certificate 


34.5 


34.1 


36.8 


Occupational/technical certificate 


13.7 


13.9 


12.5 


Associate's degree or higher 


2.2 


1.8 


4.4 


None of the above 


2.3 


2.6 


0.7 


Date of high school graduation/GED receipt (%) 


During the past year 


4.9 


4.9 


5.0 


Between 1 and 5 years ago 


28.3 


29.1 


24.3 


Between 5 and 10 years ago 


37.2 


36.7 


40.0 


More than 10 years ago 


21.1 


21.0 


21.4 


Missing 


8.5 


8.4 


9.3 


Main reason for enrolling in college b (%) 


To complete a certificate program 


2.5 


1.9 


5.9 


To obtain an associate's degree 


49.8 


49.4 


52.2 


To transfer to a 4-year college/university 


45.2 


45.7 


42.6 


To obtain/update job skills 


3.1 


3.1 


2.9 


Other 


2.0 


1.8 


2.9 


First person in family to attend college (%) 


31.1 


30.3 


35.3 


Language other than English spoken regularly in home (%) 


52.6 


47.6 


79.3 


Sample size 


882 


742 


140 



SOURCE: MDRC calculations using Baseline Information Form (BIF) data. 

NOTES: Characteristics shown in italics are calculated for a proportion of the full sample. 

Missing values are only included in variable distributions for characteristics with more than 5 percent of 
the sample missing. 

Distributions may not add to 100 percent because of rounding. 

“Respondents who said they are Hispanic and chose a race are included only in the Hispanic category. 
Respondents who are not Hispanic and chose more than one race are considered multiracial. "Other" includes 
American Indian /Alaskan Native, multiracial, and other. 

b Distributions may not add to 100 percent because categories are not mutually exclusive. 
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were less likely to be married (15.4 percent married versus 25.7 percent at Hostos) and who 
were more likely to have no children (54.3 percent versus 39.9 percent at Hostos). 

The research sample should not be considered representative of the broader student 
bodies at BMCC and Hostos, whose characteristics are shown in Table 1. As discussed, the 
research sample consists only of low-income students who were between the ages of 22 and 35, 
living away from their parents, and in need of one or more developmental education require- 
ments at the time of random assignment. A comparison of the research group with the universe 
of students meeting these target criteria is planned for a future report, after study intake is 
complete. 

Appendix Table 1 shows the same demographic characteristics as those reported in 
Table 1 for the full sample, program group, and control group. An asterisk in the last column of 
the table indicates that the percentage of program group members with that characteristic is 
significantly different from the percentage of control group members. There are a handful of 
differences between the two research groups, but no more than would be expected to occur 
randomly. 26 



Patterns of Scholarship Receipt 

A key implementation question is: What percentage of students in the program group 
earned the scholarship? Table 3 attempts to answer this question by presenting scholarship take- 
up rates for the first and second program semesters and first summer term. 27 

Since students were already registered for six or more credits when they entered the 
study, it was expected that nearly all program group students would receive the first two 
payments of the first program semester. In fact, 99.3 percent and 96.8 percent did receive the 
first and second payments, respectively. Close to 70 percent met the benchmark at the end of the 



~ h An omnibus test was conducted to assess whether overall systematic differences in baseline characteris- 
tics were observed between the two research groups. The model’s likelihood ratio test yielded a p-value of 
0.44. Convention suggests that this probability is large enough such that these potential differences can be 
ignored in the analyses. In addition, characteristics of the PBS and PBS plus summer groups were compared. 
The omnibus test yielded a p-value of 0.81 . 

27 hi this report, outcomes are presented for the first and second semesters that each sample member was in 
the study (called the “first program semester” and the “second program semester”). For example, for the fall 
2008 cohort, the first program semester is fall 2008 and the second program semester is spring 2009. For the 
spring 2009 cohort, the first program semester is spring 2009 and the second program semester is fall 2009. As 
of the writing of this report, only partial information, such as enrollment data, is available for the second 
program semester. In addition to outcomes presented for the first and second semesters that each sample 
member was in the study, outcomes are shown for the first ensuing summer term — which for the students 
covered in this report was summer 2009. 
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The Performance-Based Scholarship Demonstration 
Table 3 



Scholarship Receipt Among Program Group Members, 
Fall 2008 and Spring 2009 Cohorts 

Borough of Manhattan and Hostos Community Colleges 





Program 


Outcome 


Group 


PBS and PBS Plus Summer 




First program semester 




Received one or more scholarship payments (%) 


99.3 


Received initial payment 


99.3 


Received midterm payment 


96.8 


Received final payment 


69.4 


Average scholarship amount ($) 


1,093 


Second program semester 




Received initial scholarship payment (%) 


80.4 


Sample size 


444 


PBS Plus Summer onlv 




First sunmier term 




Received one or more scholarship payments (%) 


34.7 


Payment for enrolling in courses 


30.2 


Payment for participating in summer immersion program 


4.5 


Received initial payment 


34.7 


Received final payment 


27.9 


Received full-time scholarship 


12.6 


Received part-time scholarship 


15.3 


Average scholarship amount ($) 


799 


Sample size 


222 



SOURCE: MDRC calculations from Borough of Manhattan Community College and Hostos 
Community College scholarship payment data. 

NOTES: Characteristics shown in italics are calculated for a subset of the full sample. 

During the first program semester, 1 percent of the sample did not receive the initial and/or the 
midterm payment on the scheduled date, but recouped the appropriate payments at the end of the 
semester. In addition, 1 percent of the sample did not receive the full amount "earned" because the 
sample members owed an outstanding balance to the college. In this table, these sample members are 
counted as having received each of these payments. 



18 








semester and received the final payment, earning the full $1,300. This suggests that while 
participation in the program during the first semester was quite high, a sizeable portion of the 
sample (about 30 percent) failed to earn the full scholarship amount. However, a strong majority 
(approximately 80 percent) of the program group members enrolled in six or more credits the 
following semester, earning them the initial payment for the second program semester. 

The bottom panel of Table 3 shows that about a third of the PBS plus summer group re- 
ceived an initial payment of $200. Students were eligible for a summer scholarship if they 
enrolled in three or more credits or participated in the colleges’ summer immersion programs. 
These programs are designed to help students complete their developmental education require- 
ments by offering them intensive workshops or tutoring. At the end of the programs, students 
are able to retake the CUNY placement exams. While there are no credits associated with the 
workshops or tutoring, a credit hour proxy was assigned to each of these activities based on the 
requirement that it was trying to fill. The proxy was used to make the part-time or full-time 
scholarship determination. 

It is important to note that the majority of PBS plus summer group members did not 
enroll in courses or the summer immersion program, essentially leaving the summer scholarship 
money on the table. Many sample members may not have been able to take advantage of this 
offer because of competing family and work responsibilities. As noted, about half of the 
research sample consists of parents — and some of these parents may not have child care during 
the summer. Also, work schedules may not have permitted students to take one or more summer 
courses, especially since these courses, with their compressed time frames, typically require 
more hours per week than regular courses. For example, summer courses at BMCC and Hostos 
can require two or three times as many in-class hours per week as the same courses offered 
during the fall or spring semesters. 

Most of the members of the PBS plus summer group who earned the initial payment al- 
so received the final payment. As shown in Table 3, almost 35 percent of the members of the 
PBS plus summer group earned the initial payment, while almost 28 percent of that same group 
received the final payment: About 13 percent of members of the PBS plus summer group 
received the remainder of the full-time scholarship, giving them a total of $1,300 for the 
summer. The rest of those who received the final payment (or about 15 percent of members of 
the PBS plus summer group) earned the final part-time scholarship payment, giving them a total 
of $650 for the summer. 



An Early Look at Program Impacts 

As discussed, a key goal of the PBS Demonstration is to test whether performance- 
based scholarships help students perform better in college. During the first program semester, 
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the scholarship may have induced program group members to complete more credits and earn 
higher GPAs than control group members. There may also have been increases in persistence in 
the second program semester, as well as an increase in credit accumulation. This section 
presents impacts on these early academic outcomes that might have been affected by the 
program. 

First Program Semester 

The top panel of Table 4 shows the results for the first program semester. The first col- 
umn of the table shows outcomes for the program group, the second column shows outcomes 
for the control group, and the third column shows the difference between the two groups, or the 
impact of the program. The asterisks are used to designate differences that are statistically 
significant, or not likely due to chance. The last column shows the standard error of the impact 
estimate. 

During the first program semester, nearly all members of both the program and control 
groups registered for one or more courses. This is not surprising because registering for at least 
six credits was one of the eligibility criteria for random assignment. 

As shown in the second row of Table 4, members of the program group were more like- 
ly to be enrolled full time (taking 12 or more credits) than control group members — 76.4 
percent versus 70.8 percent — yielding a statistically significant 5.6 percentage point difference. 
This difference was somewhat unexpected since the students in both groups had created their 
schedules before they entered the study. Part of this impact might have been driven by students 
in the program group deciding to add an extra course or choosing to keep a course they might 
have otherwise dropped — with the extra money from the scholarship permitting them to pay 
for an additional course or to work fewer hours, freeing up more time for school. Program 
group students may have also thought there was an advantage to enrolling in more credits to 
increase their odds of earning six credits with the required grade of “C” or better. 

However, for about half of the sample, the impact on full-time enrollment rates is very 
likely attributable to factors others than the ones just mentioned. This is because approximately 
half of all sample members were not informed whether they were in the program or control 
groups — and thus not informed about whether they would or would not be eligible for the 
performance-based scholarship — until after the add/drop period had ended. Researchers will 
continue to examine this outcome in a future report that will draw on enrollment data for the full 
sample. 



Tied to the increase in full-time enrollment, the program group, on average, attempted 
0.5 more credits than the control group. Importantly, the program group also earned 0.6 credits 
more than the control group — an outcome that suggests that program group members were 
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The Performance-Based Scholarship Demonstration 
Table 4 

Academic Outcomes Among Sample Members: First and Second Program 
Semesters, Fall 2008 and Spring 2009 Cohorts 



Borough of Manhattan and Hostos Community Colleges 





Program 


Control 






Standard 


Outcome 


Group 


Group 


Difference 




Error 


First program semester 












Enrolled as of semester end (%) 


95.7 


95.7 


0.1 




1.4 


Full time 3 


76.4 


70.8 


5.6 


* 


3.0 


Part time b 


18.7 


24.7 


-6.0 


** 


2.8 


Average number of credits attempted 


12.5 


12.1 


0.5 


* 


0.3 


Regular credits 


7.9 


7.6 


0.3 




0.3 


Developmental credits 


4.6 


4.5 


0.1 




0.3 


Average number of credits earned 


8.6 


8.1 


0.6 


* 


0.3 


Regular credits 


6.3 


5.9 


0.4 




0.3 


Developmental credits 


2.3 


2.2 


0.2 




0.2 


Number of credits earned (%) 












12 or more 


36.3 


31.5 


4.7 




3.2 


6 to less than 12 


39.4 


41.8 


-2.4 




3.3 


0 to less than 6 


24.3 


26.7 


-2.4 




2.9 


Term GPA (%) 












3.0 to 4.0 


36.9 


38.6 


-1.6 




3.3 


2.0 to 2.9 


31.1 


28.3 


2.8 




3.1 


Less than 2.0 


20.5 


20.3 


0.2 




2.7 


No GPA C 


11.5 


12.8 


-1.3 




2.2 


Withdrew from any courses (%) 


19.8 


22.1 


-2.3 




2.7 


Number of course withdrawals 


0.3 


0.3 


0.0 




0.0 


Second program semester 












Enrolled as of semester end (%) 


77.5 


76.2 


1.2 




2.8 


Full time 3 


59.9 


52.5 


7.4 


** 


3.3 


Part time b 


15.8 


21.9 


-6.1 


** 


2.6 


Average number of credits attempted 


9.8 


9.3 


0.6 




0.4 


Regular credits 


7.4 


7.0 


0.4 




0.3 


Developmental credits 


2.4 


2.3 


0.1 




0.2 


Sample size (total = 882) 


444 


438 








(continued) 
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Table 4 (continued) 



SOURCE: MDRC calculations from Borough of Manhattan Community College and Hostos 
Community College transcript data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by research cohort and college. 

Students who dropped or withdrew from all of their courses are not considered registered. Their 
credits and grades are not included in the outcomes shown in this table. 

A few regular courses have excess "compensatory credits" associated with them. These credits are 
used to calculate total credit hours, but do not count toward a degree. In this table, compensatory credits 
are included in the developmental credits category and make up less than 1 percent of those credits. 
“Full-time enrollment is defined as 12 or more credits. 
b Part-time enrollment is defined as 6 credits to less than 12 credits. 

c The "No GPA" category includes students who are not enrolled and students who took only 
noncredit remedial courses, which are not included in GPA calculations. 

able to manage their increased workloads. Both the difference in the average rates of credits 
attempted and the difference in the average number of credits earned were statistically signifi- 
cant. 



An expected outcome of the program was an increase in term GPA. At this point, there 
was no significant difference between the program and control groups on this measure. 

One question about the performance-based scholarship is whether it would encourage 
program group members to withdraw from courses at differing rates from those of control group 
members. It is plausible that some students in the program group would have devised strategies 
for earning the scholarship that would alter their course loads — for example, taking more 
courses to hedge their bets or withdrawing from some courses in order to focus their energies on 
fewer courses. However, the scholarship did not appear to influence the rate of withdrawal for 
the first two cohorts, since about the same proportion (less than a quarter) of both the program 
group and the control group withdrew from one or more courses. 

Second Program Semester 

The second panel of Table 4 presents registration information for the second program 
semester. Grade information for this semester was not available at the time of this writing. 

As shown in the table, just under 80 percent of both groups registered for one or more 
courses; thus, the enrollment rates for the two groups were essentially the same. But as also 
shown in the table, the program group was more likely to enroll full time. There is a statistically 
significant difference of 7.4 percentage points, which represents a 14 percent increase in full- 
time enrollment. 
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First Summer Term 

As mentioned, half of the program group members were eligible to receive a summer 
scholarship. To understand how adding a summer scholarship might have influenced a student’s 
summer course-taking patterns and academic performance, an analysis was conducted to look at 
these outcomes separately for those in the PBS plus summer, PBS, and control groups. The 
results are presented in Table 5. Because this analysis is based on half of the report sample (the 
first two cohorts) and is broken down into three groups, the sample sizes are quite small and 
findings should be interpreted with caution. 

The first three columns of this table present average outcome levels for the three re- 
search groups: the PBS plus summer group, the PBS group, and the control group. These 
columns are followed by columns that present comparisons of outcomes for these three groups. 
Each comparison answers a specific question. Comparing outcomes for the PBS plus summer 
group and the control group addresses the question: What were the effects of a combined 
regular-year and summer scholarship program on summer semester attendance? Comparing 
outcomes for the PBS group and the control group helps to answer the question: What were the 
effects of a regular-year scholarship by itself on summer attendance? Finally, comparing the 
PBS plus summer group and PBS group outcomes addresses the question: What were the added 
effects of the summer scholarship on summer attendance? 

The first comparison looks at the PBS plus summer group versus the control group. 
About 28 percent of the PBS plus summer group registered for summer courses compared with 
19.6 percent of the control group, yielding a statistically significant difference of 7.8 percentage 
points. This represents a 40 percent increase in summer enrollment. There is also a significant 
difference of about half a credit on credits attempted and earned. 

The second comparison looks at the PBS group versus the control group. There are no 
statistically significant differences between outcomes for these groups during the first summer 
term. The final comparison presents the differences between the PBS plus summer and PBS 
groups, and again there are some significant differences between outcomes for the PBS summer 
group and outcomes for the comparison group — which in this case is the PBS group. Members 
of the PBS plus summer group enrolled at higher rates and attempted more credits than mem- 
bers of the PBS group. The impact on enrollment for these two groups is 6.7 percentage points, 
which is a 32 percent increase in rates of summer enrollment. 

This analysis suggests that offering a summer scholarship can increase summer enroll- 
ment rates and the number of credits earned. However, if no scholarship for summer school is 
offered, earning a performance-based scholarship during the fall and/or spring semesters does 
not appear to have an effect on school attendance during the summer. These outcomes will be 
further examined in a future report that covers the full sample for the New York program. 
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The Performance-Based Scholarship Demonstration 
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Conclusion and Next Steps 

This early look at the PBS demonstration in New York suggests that the program had a 
few modest effects, including increased full-time enrollment during both program semesters and 
increased credit attainment in the first program semester. The summer scholarship had a 
sizeable impact on rates of summer registration and credits earned. Given the limited follow-up 
period for this report, it will be interesting to see whether these early differences translate into 
long-term academic gains, such as higher rates of persistence and faster progress toward degree 
attainment. 

A research report of longer-term findings will be released in early 2012. The report will 
contain findings from an implementation study, the impact study, and a cost study. The imple- 
mentation study will describe how the performance-based scholarship program operates, 
identify promising recruitment strategies and other best practices, and capture the experiences 
and insights of students and administrators by analyzing information from focus groups and 
interviews. The impact study will extend the analyses presented in this report by looking at 
outcomes from the full sample and evaluating alternative explanations for the findings using 
data from administrative records (including data on patterns of student retention and credit 
accumulation, and on graduation and transfer rates) and from a student survey that will cover 
such topics as time spent studying, motivation, and use of scholarship monies. The cost study 
will provide a systematic analysis of the cost of operating the program. 
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Appendix A 

Selected Characteristics of Sample Members at Baseline 




The Performance-Based Scholarship Demonstration 
Appendix Table A.l 

Selected Characteristics of Sample Members at Baseline, by Research Group, 
Fall 2008 and Spring 2009 Cohorts 



Borough of Manhattan and Hostos Community Colleges 



Characteristic 


Full 

Sample 


Program 

Group 


Control 

Group 


Gender (%) 


Male 


30.8 


31.3 


30.3 


Female 


69.2 


68.7 


69.7 


Age (%) 


22-26 years 


54.8 


54.3 


55.3 


27-30 years 


26.1 


25.9 


26.3 


3 1 years and over 


19.2 


19.8 


18.5 


Average age (years) 


26.7 


26.7 


26.7 


Marital status (%) 


Married 


17.0 


15.8 


18.3 


Unmarried 


75.3 


11.0 


73.5 


Missing 


7.7 


7.2 


8.2 


Race/ethnicity a (%) 


Hispanic 


43.9 


43.6 


44.2 


White 


7.1 


8.2 


6.0 


Black 


37.1 


36.6 


37.7 


Asian or Pacific Islander 


9.2 


9.3 


9.1 


Other 


2.7 


2.3 


3.0 


Number of children (%) 


0 


52.1 


55.2 


48.9 * 


1 


25.7 


22.7 


28.8 ** 


2 


14.3 


14.3 


14.3 


3 or more 


7.9 


7.7 


8.1 


Among sample members with children 


Average age of youngest child (years) 


4.3 


4.1 


4.4 


Household receiving any government benefits (%) 


35.8 


35.3 


36.3 


Missing 


12.1 


12.6 


11.6 


Financially dependent on parents (%) 


1.4 


1.3 


1.4 


Missing 


5.4 


5.6 


5.3 


Currently employed (%) 


56.8 


56.6 


56.9 



(continued) 
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Appendix Table A.l (continued) 



Characteristic 


Full 

Sample 


Program 

Group 


Control 

Group 


Among those currently employed 

Number of hours worked per week in current job (%) 


1-10 hours 


6.7 


6.3 


7.0 


11-20 hours 


24.4 


27.8 


21.0 


21-30 hours 


21.0 


22.3 


19.8 


31-40 hours 


44.8 


40.6 


48.9 


More than 40 hours 


3.1 


3.0 


3.3 


Average hourly wage at current job ($) 


11.2 


11.0 


11.5 


Highest grade completed (%) 


10th grade or lower 


16.9 


18.5 


15.3 


11th grade 


12.9 


11.9 


13.9 


12 th grade 


64.9 


64.2 


65.5 


Missing 


5.3 


5.4 


5.3 


Diplomas/degrees eamed b (%) 


High school diploma 


64.1 


63.3 


65.0 


General Educational Development (GED) certificate 


34.5 


34.9 


34.1 


Occupational/technical certificate 


13.7 


11.6 


15.8 * 


Associate's degree or higher 


2.2 


2.3 


2.1 


None of the above 


2.3 


2.3 


2.3 


Date of high school graduation/GED receipt (%) 


During the past year 


4.9 


3.6 


6.2 * 


Between 1 and 5 years ago 


28.3 


31.5 


25.1 ** 


Between 5 and 10 years ago 


37.2 


35.4 


39.0 


More than 10 years ago 


21.1 


21.4 


20.8 


Missing 


8.5 


8.1 


8.9 


Main reason for enrolling in college 13 (%) 


To complete a certificate program 


2.5 


3.0 


2.1 


To obtain an associate's degree 


49.8 


49.5 


50.1 


To transfer to a 4-year college/university 


45.2 


46.1 


44.3 


To obtain/update job skills 


3.1 


3.0 


3.2 


Other 


2.0 


1.4 


2.6 


First person in family to attend college (%) 


31.1 


32.7 


29.4 


Language other than English spoken regularly in home (%) 


52.6 


51.2 


54.1 


Sample size 


882 


444 


438 



(continued) 
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Appendix Table A.l (continued) 

SOURCE: MDRC calculations using Baseline Information Form (BIF) data. 

NOTES: To analyze whether baseline characteristics jointly predicted research group status, a likelihood ratio 
test was performed. This yielded a p-value of 0.44. Convention suggests that these probabilities are large enough 
that these potential differences can be ignored in the analyses. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by research cohort and site. 

Characteristics shown in italics indicate nonexperimental data. Significance tests are not calculated for 
nonexperimental data. 

Missing values are only included in variable distributions for characteristics with more than 5 percent of the 
sample missing. 

Distributions may not add to 100 percent because of rounding. 

“Respondents who said they are Hispanic and chose a race are included only in the Hispanic category. 
Respondents who are not Hispanic and chose more than one race are considered multiracial. "Other" includes 
American Indian /Alaskan Native, multiracial, and other. 

b Distributions may not add to 100 percent because categories are not mutually exclusive. 
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About MDRC 



MDRC is a nonprofit, nonpartisan social and education policy research organization dedicated 
to learning what works to improve the well-being of low-income people. Through its research 
and the active communication of its findings, MDRC seeks to enhance the effectiveness of so- 
cial and education policies and programs. 

Founded in 1974 and located in New York City and Oakland, California, MDRC is best known 
for mounting rigorous, large-scale, real-world tests of new and existing policies and programs. 
Its projects are a mix of demonstrations (field tests of promising new program approaches) and 
evaluations of ongoing government and community initiatives. MDRC’s staff bring an unusual 
combination of research and organizational experience to their work, providing expertise on the 
latest in qualitative and quantitative methods and on program design, development, implementa- 
tion, and management. MDRC seeks to learn not just whether a program is effective but also 
how and why the program’s effects occur. In addition, it tries to place each project’s findings in 
the broader context of related research — in order to build knowledge about what works across 
the social and education policy fields. MDRC’s findings, lessons, and best practices are proac- 
tively shared with a broad audience in the policy and practitioner community as well as with the 
general public and the media. 

Over the years, MDRC has brought its unique approach to an ever-growing range of policy 
areas and target populations. Once known primarily for evaluations of state welfare-to-work 
programs, today MDRC is also studying public school reforms, employment programs for ex- 
offenders and people with disabilities, and programs to help low-income students succeed in 
college. MDRC’s projects are organized into five areas: 

• Promoting Family Well-Being and Children’s Development 

• Improving Public Education 

• Raising Academic Achievement and Persistence in College 

• Supporting Low-Wage Workers and Communities 

• Overcoming Barriers to Employment 

Working in almost every state, all of the nation’s largest cities, and Canada and the United 
Kingdom, MDRC conducts its projects in partnership with national, state, and local govern- 
ments, public school systems, community organizations, and numerous private philanthropies. 




